You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-19 - 06:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot7s.osadl.org (updated Fri Apr 19, 2024 00:44:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
152352824460,10sleep10-21swapper/119:05:281
1524162991121,15cyclictest141rcu_preempt23:28:182
163466321050,10sleep31634654-21turbostat00:25:013
156073721010,7sleep31753-21gnome-shell20:50:313
16234772970,6sleep20-21swapper/223:50:262
152395729659,9sleep30-21swapper/319:09:213
16013902950,5chrt0-21swapper/322:50:013
152416299952,7cyclictest0-21swapper/221:32:232
152376729559,29sleep00-21swapper/019:06:570
1524162999313,17cyclictest881251-21Softwar~cThread22:45:142
1524157999385,5cyclictest0-21swapper/123:50:461
16305742910,5sleep00-21swapper/000:10:290
1524162999112,17cyclictest881251-21Softwar~cThread23:44:462
152415799912,87cyclictest0-21swapper/123:22:171
152415299915,15cyclictest141rcu_preempt22:11:400
1524157999084,4cyclictest0-21swapper/123:19:181
1524157999083,4cyclictest0-21swapper/123:28:171
1524157999083,4cyclictest0-21swapper/121:24:541
1524157999082,5cyclictest0-21swapper/120:41:091
1524157999082,5cyclictest0-21swapper/100:17:361
1524157999080,6cyclictest1615611-21/usr/sbin/munin23:30:161
1524157999080,6cyclictest1615611-21/usr/sbin/munin23:30:151
1524162998911,15cyclictest881251-21Softwar~cThread23:32:132
1524162998911,15cyclictest881251-21Softwar~cThread23:32:132
1524157998981,5cyclictest0-21swapper/122:38:541
1524157998981,5cyclictest0-21swapper/121:09:251
152396628969,14sleep20-21swapper/219:09:282
152416299888,62cyclictest0-21swapper/223:36:002
1524157998881,4cyclictest0-21swapper/120:33:191
1524157998880,5cyclictest0-21swapper/121:18:221
1524166998714,6cyclictest141rcu_preempt21:00:353
152415799879,72cyclictest881251-21Softwar~cThread00:02:081
152416699865,15cyclictest141rcu_preempt21:36:393
152416699862,11cyclictest1753-21gnome-shell23:30:093
152416699862,11cyclictest1753-21gnome-shell23:30:093
1524162998616,7cyclictest141rcu_preempt21:30:012
1524162998616,7cyclictest141rcu_preempt21:30:012
1524157998679,4cyclictest0-21swapper/123:40:181
1524157998678,5cyclictest0-21swapper/122:14:061
1524157998676,7cyclictest0-21swapper/122:18:521
152415799852,78cyclictest1753-21gnome-shell22:35:011
152415799852,40cyclictest1753-21gnome-shell21:05:001
152416299844,15cyclictest141rcu_preempt22:20:032
152416299843,27cyclictest1564077-21python321:00:322
152416299842,11cyclictest881251-21Softwar~cThread23:46:362
1524157998476,5cyclictest0-21swapper/123:55:181
152415799842,40cyclictest1753-21gnome-shell22:05:131
152415799842,40cyclictest1753-21gnome-shell19:10:341
1524166998325,52cyclictest42-21ksoftirqd/322:44:503
1524166998325,52cyclictest42-21ksoftirqd/322:44:503
152416699831,60cyclictest1753-21gnome-shell00:00:363
152416299832,19cyclictest0-21swapper/222:07:082
152416299832,16cyclictest141rcu_preempt22:40:282
152416299832,16cyclictest141rcu_preempt22:40:282
1524157998376,4cyclictest0-21swapper/120:48:011
152415299832,16cyclictest141rcu_preempt21:34:320
162399820,6rtkit-daemon0-21swapper/120:57:501
152416699823,16cyclictest0-21swapper/322:30:353
152416699822,22cyclictest1753-21gnome-shell21:12:503
152416699822,15cyclictest141rcu_preempt23:36:413
152416699822,15cyclictest141rcu_preempt21:18:163
152416299824,15cyclictest141rcu_preempt21:35:152
152416299822,8cyclictest0-21swapper/221:10:242
152416299822,15cyclictest141rcu_preempt23:24:452
152416299822,15cyclictest141rcu_preempt22:04:582
152416299822,14cyclictest141rcu_preempt19:15:112
1524157998275,4cyclictest0-21swapper/100:09:221
152415799822,69cyclictest1753-21gnome-shell20:51:351
152416699812,22cyclictest1753-21gnome-shell22:38:043
152416699812,15cyclictest141rcu_preempt20:37:343
152416699812,15cyclictest141rcu_preempt20:37:343
152416299816,8cyclictest141rcu_preempt20:44:232
152416299812,22cyclictest1753-21gnome-shell22:30:222
152416299812,21cyclictest0-21swapper/223:59:132
152416299812,15cyclictest141rcu_preempt20:52:182
152416299811,15cyclictest141rcu_preempt20:36:382
152416299811,15cyclictest141rcu_preempt20:36:382
152416299811,15cyclictest141rcu_preempt19:15:002
1524157998175,4cyclictest0-21swapper/122:27:151
1524157998173,5cyclictest0-21swapper/120:39:501
1524157998173,5cyclictest0-21swapper/120:39:491
162399800,6rtkit-daemon0-21swapper/119:24:051
162399800,6rtkit-daemon0-21swapper/119:24:041
152416699803,15cyclictest141rcu_preempt23:26:293
152416699803,13cyclictest1753-21gnome-shell20:44:593
152416699802,17cyclictest141rcu_preempt21:30:103
152416699801,18cyclictest141rcu_preempt19:10:293
152416299802,8cyclictest0-21swapper/200:00:492
1524162998023,8cyclictest34-21ksoftirqd/200:15:512
152416299802,21cyclictest0-21swapper/221:22:052
152416299802,15cyclictest141rcu_preempt20:55:562
152416299802,14cyclictest141rcu_preempt00:10:432
152416299801,14cyclictest141rcu_preempt00:06:482
1524157998073,5cyclictest0-21swapper/122:47:541
1524157998073,4cyclictest0-21swapper/123:49:181
1524152998062,7cyclictest13-21ksoftirqd/023:50:230
1524152998062,13cyclictest1753-21gnome-shell23:40:020
152415299801,9cyclictest1627904-21latency_hist00:05:010
152416699793,15cyclictest141rcu_preempt00:15:323
1524166997923,8cyclictest42-21ksoftirqd/320:46:583
152416699792,15cyclictest141rcu_preempt23:20:443
152416699792,15cyclictest141rcu_preempt22:50:113
152416699792,14cyclictest141rcu_preempt21:09:253
152416699791,19cyclictest1753-21gnome-shell23:50:253
152416699791,15cyclictest141rcu_preempt23:45:273
152416299792,8cyclictest0-21swapper/200:20:112
152416299792,27cyclictest0-21swapper/221:05:272
152416299792,15cyclictest141rcu_preempt21:16:532
152415799792,72cyclictest1634041-21sensors_temp00:20:261
1524152997963,12cyclictest1753-21gnome-shell00:15:570
1524152997961,13cyclictest13-21ksoftirqd/020:42:510
152416699786,11cyclictest141rcu_preempt22:25:313
152416699783,14cyclictest141rcu_preempt20:55:033
152416699783,13cyclictest1753-21gnome-shell21:26:403
152416699783,13cyclictest1753-21gnome-shell21:26:393
152416699782,12cyclictest141rcu_preempt21:21:273
152416299782,7cyclictest0-21swapper/223:18:352
152416299782,7cyclictest0-21swapper/220:46:112
152416299782,58cyclictest0-21swapper/222:23:442
152416299782,20cyclictest1753-21gnome-shell22:26:112
152416299782,16cyclictest141rcu_preempt22:35:192
152416299782,16cyclictest141rcu_preempt22:10:182
152416299782,15cyclictest0-21swapper/220:34:082
1524157997870,5cyclictest0-21swapper/122:03:201
152415799782,33cyclictest881251-21Softwar~cThread00:14:021
152415799782,33cyclictest880726-21firefox21:10:431
152416699771,14cyclictest141rcu_preempt00:10:113
1524157997769,5cyclictest0-21swapper/121:33:441
1524157997755,18cyclictest26-21ksoftirqd/121:30:011
1524157997755,18cyclictest26-21ksoftirqd/121:30:001
152415799772,70cyclictest1601950-21/usr/sbin/munin22:50:201
152415799772,69cyclictest880726-21firefox23:39:541
15532342760,4sleep30-21swapper/320:30:223
152416699763,48cyclictest1753-21gnome-shell23:56:253
152416699762,15cyclictest141rcu_preempt22:05:173
152416699761,14cyclictest141rcu_preempt00:09:293
152416299761,14cyclictest141rcu_preempt22:50:122
1524157997668,5cyclictest0-21swapper/119:48:191
152415799762,68cyclictest1598461-21diskstats22:40:171
152415799762,68cyclictest1598461-21diskstats22:40:161
1524152997665,7cyclictest1753-21gnome-shell20:52:510
152416699759,6cyclictest141rcu_preempt23:41:523
1524157997547,13cyclictest26-21ksoftirqd/122:20:311
1524152997563,7cyclictest1753-21gnome-shell21:14:310
152416699744,14cyclictest141rcu_preempt22:21:563
152416699742,68cyclictest1753-21gnome-shell22:15:393
152416699742,15cyclictest141rcu_preempt22:10:033
162399730,5rtkit-daemon0-21swapper/121:36:091
1524152997362,7cyclictest1753-21gnome-shell21:36:170
1524152997323,7cyclictest13-21ksoftirqd/019:14:480
1524152997254,5cyclictest13-21ksoftirqd/022:20:110
1524152997120,46cyclictest1753-21gnome-shell00:06:010
1524152997113,46cyclictest1753-21gnome-shell21:20:300
152416699703,15cyclictest141rcu_preempt19:15:073
1524152997046,7cyclictest13-21ksoftirqd/023:35:080
152416699692,45cyclictest881251-21Softwar~cThread22:04:283
152415799692,43cyclictest1609447-21ntp_states23:10:261
1524152996959,6cyclictest13-21ksoftirqd/022:31:350
1524152996951,13cyclictest13-21ksoftirqd/022:40:130
1524152996951,13cyclictest13-21ksoftirqd/022:40:120
152415299692,21cyclictest1753-21gnome-shell00:21:350
1524152996916,48cyclictest1753-21gnome-shell22:08:590
1524157996827,34cyclictest1526328-21df_inode19:15:051
1524152996854,5cyclictest13-21ksoftirqd/023:55:530
1524152996839,20cyclictest1753-21gnome-shell20:33:190
1524152996812,51cyclictest881251-21Softwar~cThread22:38:220
152415299681,13cyclictest1069-21runrttasks23:32:370
152415299681,13cyclictest1069-21runrttasks23:32:360
1524152996756,7cyclictest1753-21gnome-shell21:18:330
152415299672,22cyclictest1753-21gnome-shell23:49:190
152415299672,22cyclictest1753-21gnome-shell20:37:010
152415299672,22cyclictest1753-21gnome-shell20:37:010
1524152996718,45cyclictest1753-21gnome-shell22:18:220
1524152996717,46cyclictest1753-21gnome-shell22:51:560
152415299662,19cyclictest1753-21gnome-shell22:03:390
1524152996617,45cyclictest1753-21gnome-shell21:00:360
1524152996616,46cyclictest1753-21gnome-shell23:25:490
1524152996616,46cyclictest1753-21gnome-shell23:20:080
1524152996615,46cyclictest1753-21gnome-shell21:08:410
1524152996554,7cyclictest1753-21gnome-shell21:28:280
1524152996554,7cyclictest1753-21gnome-shell21:28:270
152415299652,21cyclictest1753-21gnome-shell22:48:190
152415299652,20cyclictest1753-21gnome-shell22:29:090
152415299652,19cyclictest1753-21gnome-shell20:56:070
152415799641,57cyclictest881251-21Softwar~cThread20:09:381
152416699632,54cyclictest1753-21gnome-shell19:48:193
152415299632,18cyclictest1753-21gnome-shell20:48:210
1524166996213,45cyclictest880726-21firefox23:18:333
1524152996152,5cyclictest0-21swapper/023:18:590
152415799602,31cyclictest1638129-21munin-run00:35:011
152415799592,30cyclictest1635280-21df00:25:161
152415299593,50cyclictest1753-21gnome-shell19:15:050
152415299592,51cyclictest1753-21gnome-shell21:40:030
152415799583,50cyclictest1753-21gnome-shell21:40:041
152415799581,32cyclictest1639261-21python300:35:311
162399570,9rtkit-daemon1622-21rtkit-daemon19:25:570
152416299563,42cyclictest880726-21firefox19:48:192
152415799566,29cyclictest1606887-21/usr/sbin/munin23:05:211
152415799561,32cyclictest1541108-21/usr/sbin/munin19:55:201
162399550,8rtkit-daemon1622-21rtkit-daemon19:38:351
15508052550,7sleep21358-1kworker/u9:120:25:022
162399540,8rtkit-daemon1622-21rtkit-daemon20:10:560
162399540,8rtkit-daemon1622-21rtkit-daemon20:10:560
162399540,8rtkit-daemon1622-21rtkit-daemon00:30:160
1524157995318,30cyclictest1530555-21cut19:25:211
162399520,8rtkit-daemon1622-21rtkit-daemon20:27:410
162399520,8rtkit-daemon1622-21rtkit-daemon20:06:480
152415299520,48cyclictest1622-21rtkit-daemon20:01:530
16353122510,20sleep234-21ksoftirqd/200:25:202
162399510,42rtkit-daemon1622-21rtkit-daemon19:48:080
152416299512,8cyclictest0-21swapper/221:40:092
162399500,42rtkit-daemon856-21gmain20:25:231
152416699502,7cyclictest0-21swapper/321:40:053
152416299508,6cyclictest141rcu_preempt20:04:592
152415799503,42cyclictest1536988-21latency_hist19:45:011
152415799502,43cyclictest1753-21gnome-shell19:50:231
1524157995017,28cyclictest1545900-21cat20:10:131
1524157995017,28cyclictest1545900-21cat20:10:121
1524157995017,28cyclictest1543874-21date20:05:021
1524157995017,27cyclictest1605217-21cstates23:00:111
1524157995017,27cyclictest1605217-21cstates23:00:111
152415799501,31cyclictest1604362-21timerwakeupswit22:55:321
152415799491,30cyclictest1550478-21Indexed~#20543720:23:211
1524166994824,20cyclictest1541992-21unin-run20:00:013
152416299487,37cyclictest1753-21gnome-shell19:59:332
1524157994816,27cyclictest1579143-21grep21:45:121
1524157994816,27cyclictest1547724-21expr20:15:141
152415799481,32cyclictest1753-21gnome-shell19:34:061
16394562470,13sleep013-21ksoftirqd/000:35:320
1524157994719,23cyclictest1583217-21ntpq21:55:271
15401152460,6sleep21527722-21kworker/u8:019:53:462
1524166994613,10cyclictest141rcu_preempt23:10:023
1524166994510,5cyclictest0-21swapper/323:12:243
152415799452,26cyclictest1581214-21irqstats21:50:211
1524152994528,12cyclictest881251-21Softwar~cThread19:58:250
1524166994421,18cyclictest42-21ksoftirqd/300:30:293
1524166994410,15cyclictest1753-21gnome-shell19:36:323
152416299442,17cyclictest0-21swapper/223:00:292
152416299442,17cyclictest0-21swapper/223:00:282
1524166994326,12cyclictest1545431-21munin-run20:10:013
152416299422,17cyclictest0-21swapper/219:35:252
1524166994126,11cyclictest1604769-21awk23:00:003
1524166994116,20cyclictest881251-21Softwar~cThread23:01:213
1524166994116,20cyclictest881251-21Softwar~cThread23:01:213
152416699411,36cyclictest1753-21gnome-shell19:54:443
152416299412,34cyclictest1753-21gnome-shell00:34:002
1524166994029,7cyclictest1550064-21cat20:20:313
1524166994026,11cyclictest42-21ksoftirqd/319:30:203
1524166994022,13cyclictest881343-21Timer21:51:333
1524166994020,16cyclictest1753-21gnome-shell19:25:143
1524162994026,10cyclictest1536882-21ls19:45:022
152416299402,34cyclictest1532588-21wc19:30:312
152416299402,18cyclictest141rcu_preempt21:50:022
1524166993922,13cyclictest1583208-21ntpq21:55:253
152416699392,10cyclictest0-21swapper/300:25:243
152416299399,6cyclictest141rcu_preempt23:15:002
152416299391,10cyclictest0-21swapper/223:09:242
1524152993924,11cyclictest881251-21Softwar~cThread21:51:210
1524152993923,11cyclictest438-20systemd-journal23:07:510
1524166993823,11cyclictest1579573-21load21:45:213
1524166993822,12cyclictest1638790-21smartctl00:35:193
1524166993822,12cyclictest1541948-21cron20:04:553
1524166993822,11cyclictest1528317-21dpkg19:20:123
1524166993822,11cyclictest1528317-21dpkg19:20:123
152416299383,15cyclictest141rcu_preempt22:59:592
1524162993824,10cyclictest1583656-21seq21:56:452
1524162993821,13cyclictest34-21ksoftirqd/220:15:482
1524152993813,12cyclictest1582429-21date21:55:020
1524166993723,10cyclictest1551232-21ls20:25:143
152416699372,30cyclictest881251-21Softwar~cThread19:40:163
152416699371,17cyclictest1753-21gnome-shell20:19:513
152416299378,7cyclictest141rcu_preempt20:06:002
1524162993725,8cyclictest1753-21gnome-shell00:39:362
1524162993722,11cyclictest34-21ksoftirqd/220:14:032
1524162993722,11cyclictest34-21ksoftirqd/220:14:032
1524162993722,11cyclictest1530316-21expr19:25:122
1524162993719,14cyclictest1581211-21irqstats21:50:202
162399360,29rtkit-daemon1753-21gnome-shell22:55:330
1524162993617,14cyclictest881251-21Softwar~cThread19:23:412
1524162993617,14cyclictest881251-21Softwar~cThread19:23:412
152415299362,5cyclictest0-21swapper/021:48:330
1524152993625,7cyclictest0-21swapper/019:34:430
1524152993620,12cyclictest855-21dbus-daemon19:40:010
1524152993620,11cyclictest1549422-21expr20:20:150
1524152993617,14cyclictest855-21dbus-daemon23:05:010
1524152993617,14cyclictest855-21dbus-daemon23:05:010
1524166993517,13cyclictest1546128-21cat20:10:193
1524166993517,13cyclictest1546128-21cat20:10:193
152415299352,30cyclictest0-21swapper/019:40:580
1524152993521,9cyclictest880726-21firefox20:18:180
1524152993519,11cyclictest1635250-21expr00:25:150
152415299351,7cyclictest1538734-21/usr/sbin/munin19:50:310
152415299351,11cyclictest141rcu_preempt23:11:440
152416299344,26cyclictest1552501-21latency_hist20:30:012
1524152993421,9cyclictest1753-21gnome-shell19:20:120
1524152993421,9cyclictest1753-21gnome-shell19:20:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional